Deep Feedforward Neural Network Research Articles

Performance evaluation of deep learning approaches for fault diagnosis of rotational mechanical systems using vibration, sound, and acoustic emission signals

The present study emphasizes an optimized deep learning algorithm for gearbox fault detection using vibration, sound, and acoustic emission signals. Statistical and acoustic features are extracted from these signals, and various neural network algorithms are explored. The supervised deep feed forward neural network (DFFNN) demonstrates excellent performance with vibration signals but limited accuracy with sound and acoustic emission signals. To address this, unsupervised algorithms are optimized and compared with vibration-based classification. The findings show that unsupervised neural networks, particularly the auto-encoder and stacked auto-encoder architectures, achieve improved classification accuracy by leveraging the unique characteristics of acoustic emission signals. The unsupervised models also effectively overcome the vanishing gradient problem via regularization, enhancing their training efficiency. The stacked auto-encoder, with multiple layers of encoders and decoders, reduces computation time by 40% and memory consumption. These optimized algorithms hold promise for automated fault detection systems. The auto-encoder and stacked auto-encoder, utilizing vibration, sound, and acoustic emission signals, offer enhanced classification accuracy and can facilitate real-time monitoring of rotating mechanical systems. However, further optimization is needed to maximize their performance. In a nutshell, the supervised DFFNN excels in utilizing vibration signals for fault detection, while the unsupervised models exploit the distinctive characteristics of acoustic emission signals. Future research will focus on refining these algorithms to enhance their effectiveness. Implementing these optimized deep learning approaches can lead to autonomous fault detection systems, eliminating the need for continuous human supervision.

Electronic properties prediction enhancement of 36 ternary III-IB-VI alloys using a deep feed-forward neural network

The electronic properties of 36 ternary III-IB-VI monolayers have been investigated using a correlation study, in which a comprehensive study of the band structure of these alloys is provided. To this end, a combination of density functional theory calculations and a deep feed-forward neural network is conducted. Principal component analysis and data augmentation are also implemented as pre-processing. The former reduces the size of features to mitigate redundancy, and the latter can prevent overfitting. The deep feed-forward neural network is then used for the final prediction. During the study, bandgaps of the alloys are predicted indirectly and compared with directly obtained bandgaps. The results of the predictions show a maximum precision of 0.00089 for nMSE using the deep feed-forward neural network, which confirms strong dependencies between the electronic properties of the alloys.

Exploring Physics-Informed Neural Networks for the Generalized Nonlinear Sine-Gordon Equation

The nonlinear sine-Gordon equation is a prevalent feature in numerous scientific and engineering problems. In this paper, we propose a machine learning-based approach, physics-informed neural networks (PINNs), to investigate and explore the solution of the generalized non-linear sine-Gordon equation, encompassing Dirichlet and Neumann boundary conditions. To incorporate physical information for the sine-Gordon equation, a multiobjective loss function has been defined consisting of the residual of governing partial differential equation (PDE), initial conditions, and various boundary conditions. Using multiple densely connected independent artificial neural networks (ANNs), called feedforward deep neural networks designed to handle partial differential equations, PINNs have been trained through automatic differentiation to minimize a loss function that incorporates the given PDE that governs the physical laws of phenomena. To illustrate the effectiveness, validity, and practical implications of our proposed approach, two computational examples from the nonlinear sine-Gordon are presented. We have developed a PINN algorithm and implemented it using Python software. Various experiments were conducted to determine an optimal neural architecture. The network training was employed by using the current state-of-the-art optimization methods in machine learning known as Adam and L-BFGS-B minimization techniques. Additionally, the solutions from the proposed method are compared with the established analytical solutions found in the literature. The findings show that the proposed method is a computational machine learning approach that is accurate and efficient for solving nonlinear sine-Gordon equations with a variety of boundary conditions as well as any complex nonlinear physical problems across multiple disciplines.

Approximating Nonlinear Functions With Latent Boundaries in Low-Rank Excitatory-Inhibitory Spiking Networks.

Deep feedforward and recurrent neural networks have become successful functional models of the brain, but they neglect obvious biological details such as spikes and Dale's law. Here we argue that these details are crucial in order to understand how real neural circuits operate. Towards this aim, we put forth a new framework for spike-based computation in low-rank excitatory-inhibitory spiking networks. By considering populations with rank-1 connectivity, we cast each neuron's spiking threshold as a boundary in a low-dimensional input-output space. We then show how the combined thresholds of a population of inhibitory neurons form a stable boundary in this space, and those of a population of excitatory neurons form an unstable boundary. Combining the two boundaries results in a rank-2 excitatory-inhibitory (EI) network with inhibition-stabilized dynamics at the intersection of the two boundaries. The computation of the resulting networks can be understood as the difference of two convex functions and is thereby capable of approximating arbitrary non-linear input-output mappings. We demonstrate several properties of these networks, including noise suppression and amplification, irregular activity and synaptic balance, as well as how they relate to rate network dynamics in the limit that the boundary becomes soft. Finally, while our work focuses on small networks (5-50 neurons), we discuss potential avenues for scaling up to much larger networks. Overall, our work proposes a new perspective on spiking networks that may serve as a starting point for a mechanistic understanding of biological spike-based computation.

Fossil energy market price prediction by using machine learning with optimal hyper-parameters: A comparative study

Fossil energy markets are important commodities, and their price fluctuations impact worldwide economy and financial markets. Hence, it is essential to forecast the prices of fossil energy commodities. In this study, various machine learning systems are optimized by using Bayesian optimization method and implemented to forecast prices in 12 different fossil energy markets including crude oil, natural gas, propane, kerosene, gasoline, heating oil, and coal based on price information from all fossil energy markets. The optimized machine learning systems considered in modeling and simulations are Gaussian regression process, support vector regression, regression trees, k-nearest neighbor algorithm, and deep feedforward neural networks. The simulation results show that Gaussian regression process is the best system to forecast fossil energy markets. In addition, the deep learning feedforward neural networks system yields to stable forecasts. Furthermore, the prediction of the prices in natural gas, coal, and propane markets is a difficult task compared to the prediction of the prices in crude oil markets. The understanding of the effectiveness of machine learning systems in pricing different fossil energy markets can help international economic agents establish appropriate investment strategies.

Using Deep Neural Networks and Big Data to Predict Microscopic Travel Time in Work Zones

Traffic management in work zones is a challenging task that requires a balance between creating a safe environment for workers and minimizing traffic delays. Accurately predicting travel times through work zones is essential for dynamic traffic management and reducing congestion. However, complex interrupted traffic patterns in work zones (also known as construction zones) make this challenging compared with regular traffic congestion and free-flow conditions. To address this complexity, the paper develops a data-driven deep feed-forward artificial neural network to forecast under-construction travel times in work zones using an integrated data set of almost half a million observations. The variables considered in the neural network include the work zone characteristics, road design features, weather information, and traffic flow information. The raw data set comprised approximately 15 million travel time observations collected at 674 work zones. After cleaning and preprocessing the data and applying feature engineering techniques, the raw data set was reduced to 81 work zones spread along a 700 km corridor between the western borders of Alberta and Vancouver, British Columbia. The data were split into training and test data using a 90:10 ratio. Training data were used to develop a 27-input neural network with four hidden layers. After validating the test data, the neural network achieved a root mean squared error of 0.150 min and an R-squared score of 0.945, indicating a high accuracy level in estimating the under-construction travel time.

Premonition-driven deep learning model for short-term ship violent roll motion prediction based on the hull attitude premonition mechanism

Ship transit forecasting can provide advanced movement information to a multidimensional wave motion compensation system. However, traditional deep learning models based on data-driven or physical data have difficulty in accurately predicting the rapid variations in ship motion attitudes that frequently occur in complex ocean environments. Therefore, a premonition-driven deep learning model is proposed in this paper to significantly improve the prediction accuracy of ship violent roll motion. This model applies precursor information to predict premonitory values of ship prospective motion attitudes based on an integrated statistical regression model and then employs these premonitory values to revise the outputs of the deep learning time series model. First, a hull attitude premonition mechanism is presented based on biological premonition cognitive behaviour, which captures the correlation relationships between extreme values by downgrading the globally continuous time series prediction problem to a time-independent regression problem. The corresponding methods for predicting local extreme values and deep feedforward neural network cluster are established based on these correlation relationships between the extreme values to achieve precise premonitory values. Second, the time and amplitude variables of the extreme angle values are transformed to countdown time variables and sequential sets of premonitory angle values. Two types of premonition units and their respective premonitory long short-term memory neural networks are constructed to upgrade the regression problem to a globally continuous time series prediction problem. Finally, a single-step output recurrent strategy is applied to forecast the short-term ship motion attitude for fifteen seconds by considering eleven initially predicted points across the two datasets. In sixty-six simulation experiments, the mean absolute error, root mean squared error, and mean absolute percentage error of the premonition-driven long short-term memory neural networks are considerably lower than those of traditional deep learning models. These simulation experiments verify that the premonition-driven deep learning model can effectively forecast short-term violent ship roll motion and provide safeguards for cargo transfer.

Post-stack seismic inversion through probabilistic neural networks and deep forward neural networks

AbstractSeismic inversion, a nonlinear, non-unique-solution, and human-prone error problem, seeks the optimal solution in multivariate space, a task that AI algorithms perform better. So, inversion with AI becomes appropriate to extract information from post-stack data. A probabilistic neural network (PNN) and deep feedforward neural network (DFNN) inverted seismic data using multiattributes, and their results were compared with those provided by a model-based inversion (MBI). Applied to a post-stack volume of the Frontier Formation at the Teapot-Dome field, Wyoming-USA, the three methods provided porosity volumes (ϕ), gamma rays (GR), and Lamé impedances (µρ, λρ). The comparison between well-logs and the provided volumes revealed a reliable PNN prediction, while DFNN achieved the best geologically correlated parameter sections and a less reliable MBI prediction. The geomorphological time-slice analysis discriminated high ϕ-µρ areas and low GR–λρ areas, geologically interpreted as a deltaic environment, with distributary channels and beachfront lobes subject to the action of the tides, which agrees with the well-log interpretation and the basin geological evolution. The results enhanced the stacked section’s importance by the geological information contained, which would contribute to reducing the prospecting risk. Finally, PNN and DFNN encouraging results reaffirm their best searching-in-multivariate space ability.

Optimal Re-Materialization Strategies for Heterogeneous Chains: How to Train Deep Neural Networks with Limited Memory

Training in Feed Forward Deep Neural Networks is a memory-intensive operation which is usually performed on GPUs with limited memory capacities. This may force data scientists to limit the depth of the models or the resolution of the input data if data does not fit in the GPU memory. The re-materialization technique, whose idea comes from the checkpointing strategies developed in the Automatic Differentiation literature, allows data scientists to limit the memory requirements related to the storage of intermediate data (activations), at the cost of an increase in the computational cost. This paper introduces a new strategy of re-materialization of activations that significantly reduces memory usage. It consists in selecting which activations are saved and which activations are deleted during the forward phase, and then recomputing the deleted activations when they are needed during the backward phase. We propose an original computation model that combines two types of activation savings: either only storing the layer inputs, or recording the complete history of operations that produced the outputs. This paper focuses on the fully heterogeneous case, where the computation time and the memory requirement of each layer is different. We prove that finding the optimal solution is NP-hard and that classical techniques from Automatic Differentiation literature do not apply. Moreover, the classical assumption of memory persistence of materialized activations, used to simplify the search of optimal solutions, does not hold anymore. Thus, we propose a weak memory persistence property and provide a Dynamic Program to compute the optimal sequence of computations. This algorithm is made available through the Rotor software, a PyTorch plug-in dealing with any network consisting of a sequence of layers, each of them having an arbitrarily complex structure. Through extensive experiments, we show that our implementation consistently outperforms existing re-materialization approaches for a large class of networks, image sizes and batch sizes.

Hybrid Batch-Normalized Deep Feedforward Neural Network Incorporating Polynomial Regression for High-Dimensional Microwave Modeling

Hybrid Batch-Normalized Deep Feedforward Neural Network Incorporating Polynomial Regression for High-Dimensional Microwave Modeling

Topological learning in multiclass data sets.

We specialize techniques from topological data analysis to the problem of characterizing the topological complexity (as defined in the body of the paper) of a multiclass data set. As a by-product, a topological classifier is defined that uses an open subcovering of the data set. This subcovering can be used to construct a simplicial complex whose topological features (e.g., Betti numbers) provide information about the classification problem. We use these topological constructs to study the impact of topological complexity on learning in feedforward deep neural networks (DNNs). We hypothesize that topological complexity is negatively correlated with the ability of a fully connected feedforward deep neural network to learn to classify data correctly. We evaluate our topological classification algorithm on multiple constructed and open-source data sets. We also validate our hypothesis regarding the relationship between topological complexity and learning in DNN's on multiple data sets.

Physics-infused deep neural network for solution of non-associative Drucker–Prager elastoplastic constitutive model

In the present work, a physics-informed deep learning-based constitutive modeling approach has been introduced, for the first time, to solve non-associative Drucker–Prager elastoplastic solid governed by a linear isotropic hardening rule. A purely data-driven surrogate modeling approach for representing complex and highly non-linear elastoplastic constitutive response prevents accurate predictions due to the absence of prior physical information. To mitigate this, we design an efficient physics-constrained training approach leveraging prior physics-driven optimization procedures. It has been achieved by formulating a highly physics-augmented multi-objective loss function that includes elastoplastic constitutive relations, Drucker–Prager yield criterion, non-associative flow rule, Kuhn–Tucker consistency conditions, and various boundary conditions. Utilizing multiple densely connected independent feed-forward deep neural networks fed with high-fidelity numerical solutions in a data-driven loss function, the model obtains the accurate elastoplastic solution by minimizing the proposed loss function. The strength and robustness of the approach have been demonstrated by accurately solving the benchmark problem where a plastically deformed isotropic shallow stratum has been subjected to compressive pressure under plane strain Drucker–Prager yield condition. To optimize the performance and trainability of the model, extensive experiments on network architecture and various degrees of data-driven estimate shed light on significant improvement in terms of the accuracy of the elastoplastic solution, particularly, that exhibits sharp, or very localized features. Moreover, we propose a transfer learning-based PINNs modeling approach that elucidates the possibility of predicting solutions for different sets of applied stress and material parameters. Requiring significantly less training data, the framework can simultaneously enhance the accuracy of the solution and adaptability of training by demonstrating rapid convergence in critical loss components. The current study highlights a systematic development of a novel physics-informed deep learning approach which is quite generic in nature, yet robust and highly physics-augmented for transferability of known knowledge for vastly accelerated convergence with improved accuracy of predicting an accurate description of non-associative elastoplastic solution in the regime of continuum mechanics.

Combined MIMO Deep Learning Method for ACOPF with High Wind Power Integration

The higher penetration of renewable energy sources in current and future power grids requires effective optimization models to solve economic dispatch (ED) and optimal power flow (OPF) problems. Data-driven optimization models have shown promising results compared to classical algorithms because they can address complex and computationally demanding problems and obtain the most cost-effective solution for dispatching generators. This study compares the forecast performance of selected data-driven models using the modified IEEE 39 benchmark system with high penetration of wind power generation. The active and reactive power load data of each bus are generated using Monte Carlo simulations, and synthetic wind power data are generated by utilizing a physical wind turbine model and wind speed samples withdrawn from a Weibull distribution. The objective is to design and evaluate an enhanced deep learning approach for the nonlinear, nonconvex alternating current optimal power flow (ACOPF) problem. The study attempts to establish relationships between loads, generators, and bus outcomes, utilizing a multiple-input, multiple-output (MIMO) workflow. Specifically, the study compares the forecast error reduction of convolutional neural networks (CNNs), deep feed-forward neural networks (DFFNNs), combined/hybrid CNN-DFFNN models, and the transfer learning (TL) approach. The results indicate that the proposed combined model outperforms the CNN, hybrid CNN-DFFNN, and TL models by a small margin and the DFFNN by a large margin.

On the ability of standard and brain-constrained deep neural networks to support cognitive superposition: a position paper

AbstractThe ability to coactivate (or “superpose”) multiple conceptual representations is a fundamental function that we constantly rely upon; this is crucial in complex cognitive tasks requiring multi-item working memory, such as mental arithmetic, abstract reasoning, and language comprehension. As such, an artificial system aspiring to implement any of these aspects of general intelligence should be able to support this operation. I argue here that standard, feed-forward deep neural networks (DNNs) are unable to implement this function, whereas an alternative, fully brain-constrained class of neural architectures spontaneously exhibits it. On the basis of novel simulations, this proof-of-concept article shows that deep, brain-like networks trained with biologically realistic Hebbian learning mechanisms display the spontaneous emergence of internal circuits (cell assemblies) having features that make them natural candidates for supporting superposition. Building on previous computational modelling results, I also argue that, and offer an explanation as to why, in contrast, modern DNNs trained with gradient descent are generally unable to co-activate their internal representations. While deep brain-constrained neural architectures spontaneously develop the ability to support superposition as a result of (1) neurophysiologically accurate learning and (2) cortically realistic between-area connections, backpropagation-trained DNNs appear to be unsuited to implement this basic cognitive operation, arguably necessary for abstract thinking and general intelligence. The implications of this observation are briefly discussed in the larger context of existing and future artificial intelligence systems and neuro-realistic computational models.

Survival prediction of glioblastoma patients using modern deep learning and machine learning techniques

In this study, we utilized data from the Surveillance, Epidemiology, and End Results (SEER) database to predict the glioblastoma patients’ survival outcomes. To assess dataset skewness and detect feature importance, we applied Pearson's second coefficient test of skewness and the Ordinary Least Squares method, respectively. Using two sampling strategies, holdout and five-fold cross-validation, we developed five machine learning (ML) models alongside a feed-forward deep neural network (DNN) for the multiclass classification and regression prediction of glioblastoma patient survival. After balancing the classification and regression datasets, we obtained 46,340 and 28,573 samples, respectively. Shapley additive explanations (SHAP) were then used to explain the decision-making process of the best model. In both classification and regression tasks, as well as across holdout and cross-validation sampling strategies, the DNN consistently outperformed the ML models. Notably, the accuracy were 90.25% and 90.22% for holdout and five-fold cross-validation, respectively, while the corresponding R2 values were 0.6565 and 0.6622. SHAP analysis revealed the importance of age at diagnosis as the most influential feature in the DNN's survival predictions. These findings suggest that the DNN holds promise as a practical auxiliary tool for clinicians, aiding them in optimal decision-making concerning the treatment and care trajectories for glioblastoma patients.

Medium-term water consumption forecasting based on deep neural networks

Water consumption forecasting is an essential tool for water management, as it allows for efficient planning and allocation of water resources, an undervalued but indispensable resource for all living beings. With the increasing demand for accurate and timely water forecasting, traditional forecasting methods are proving to be insufficient. Deep learning techniques, which have shown remarkable performance in a wide range of applications, offer a promising approach to address the challenges of water consumption forecasting. In this work, the use of deep learning models for medium-term water consumption forecasting of residential areas is explored. A deep feed-forward neural network is developed to predict water consumption of a company’s customers for the next quarter. First, customers are grouped according to their consumption as these customers include both household consumers and special consumers such as public swimming pools, sports halls or small industries. Then, a deep feed-forward neural network is designed for household customers by obtaining the optimal values for those hyperparameters that have a great influence on the network performance. Results are reported using a real-world dataset composed of the water consumption from 1999 to 2015 on a quarterly basis, corresponding to 3262 clients of a water supply company. Finally, the proposed algorithm is evaluated by comparing it with other reference algorithms including an LSTM network.

ForecastNet Wind Power Prediction Based on Spatio-Temporal Distribution

The integration of large-scale wind power into the power grid threatens the stable operation of the power system. Traditional wind power prediction is based on time series without considering the variability between wind turbines in different locations. This paper proposes a wind power probability density prediction method based on a time-variant deep feed-forward neural network (ForecastNet) considering a spatio-temporal distribution. First, the outliers in the wind turbine data are detected based on the isolated forest algorithm and repaired through Lagrange interpolation. Then, based on the graph attention mechanism, the features of the proximity node information of the individual wind turbines in the wind farm are extracted and the input feature matrix is constructed. Finally, the wind power probability density prediction results are obtained using the ForecastNet model based on three different hidden layer variants. The experimental results show that the ForecastNet model with a hidden layer as a dense network based on the attention mechanism (ADFN) predicts better. The average width of the prediction intervals at achieved confidence levels for all interval coverage is reduced by 34.19%, 35.41%, and 35.17%, respectively, when compared to the model with the hidden layer as a multilayer perceptron. For different categories of wind turbines, ADFN also achieves relatively narrow interval average widths of 368.37 kW, 315.87 kW, and 299.13 kW, respectively.

MLBKFD: Probabilistic Model Methods to Infer Pseudo Trajectories from Single-cell Data

Cell trajectory inference is very important to understand the details of tissue cell development, state differentiation and gene dynamic regulation. However, due to the high noise and heterogeneity of the single-cell data, it is challenging to infer cell trajectory in complex biological processes. Here, we proposed a new trajectory inference method, called Metric Learning Bhattacharyya Kernel Feature Decomposition (MLBKFD). In MLBKFD, a statistical model was used to infer cell trajectory by calculating the Markov transition matrix and Bhattacharyya kernel matrix between cells. Before that, to expedite the matrix calculation in the statistical model, a deep feedforward neural network was used to perform dimensionality reduction on single-cell data. The MLBKFD was evaluated on four typical datasets as well as seven recent human fetal lung datasets. Comparisons with the two outstanding methods (i.e., DTFLOW and MARGARET) demonstrate that the MLBKFD is capable of accurately inferring cell development and differentiation trajectories from single-cell data with different sizes and sources. Notably, MLBKFD exhibits nearly twice the speed of DTFLOW while maintaining high precision, particularly when dealing with large datasets. MLBKFD provides accurate and efficient trajectory inference, empowering researchers to gain deeper insights into the complex dynamics of cell development and differentiation.

A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law

Deep Neural Networks (DNN) has transformed the automation of a wide range of industries and finds increasing ubiquity in society. The high complexity of DNN models and its widespread adoption has led to global energy consumption doubling every 3-4 months. Current energy consumption measures largely monitor system wide consumption or make linear assumptions of DNN models. The former approach captures other unrelated energy consumption anomalies, whilst the latter does not accurately reflect nonlinear computations. In this paper, we are the first to develop a bottom-up Transistor Operations (TOs) approach to expose the role of non-linear activation functions and neural network structure. As there will be inevitable energy measurement errors at the core level, we statistically model the energy scaling laws as opposed to absolute consumption values. We offer models for both feedforward DNNs and convolution neural networks (CNNs) on a variety of data sets and hardware configurations - achieving a 93.6% - 99.5% precision. This outperforms existing FLOPs-based methods and our TOs method can be further extended to other DNN models.

Extended Features Based Random Vector Functional Link Network for Classification Problem

Random vector functional link (RVFL) network has been successfully employed in diverse domains such as computer vision and machine learning, due to its universal approximation capability. Recently, the shallow RVFL architecture has been extended to deep architectures. In deep architectures, multiple hidden layers are stacked for extracting informative features from the original feature space. Therefore, having rich features, deep models are very successful compared to shallow models. In this article, we propose an extended feature RVFL (efRVFL) model that is trained over extended feature space generated analytically from the original feature space. The proposed efRVFL model has three types of features, i.e., original features, supervised randomized (newly generated) features, and unsupervised randomized features, in its feature matrix. The proposed efRVFL model with additional features has capability to capture nonlinear hidden relationships within the dataset. The proposed efRVFL model is an unstable classifier, and thus, its performance can be improved further via ensemble learning. Ensemble models are stable and accurate and have better generalization performance than single models. Therefore, we also propose an ensemble of extended feature RVFL (en-efRVFL) model. Each base model of en-efRVFL is trained over different feature spaces so that more accurate and diverse base models can be generated. The outcome of the base models is integrated via average voting scheme. Empirical evaluation over <inline-formula> <tex-math notation="LaTeX">$46$</tex-math> </inline-formula> UCI classification datasets demonstrates that the proposed efRVFL and en-efRVFL models have better performance than RVFL and other given deep models. Furthermore, the experimental results over <inline-formula> <tex-math notation="LaTeX">$12$</tex-math> </inline-formula> sparse datasets show that the proposed en-efRVFL model has a winning performance among several deep feedforward neural networks (FNNs).